System and method for improved electronic assisted medical procedures

ABSTRACT

Methods and systems for displaying an image of a medical procedure (e.g., an intraoperative image) with additional information (e.g., data) that can augment the image of the medical procedure are provided. Augmenting the image can include overlaying data from a first image to a second image. Overlaying the data can involve determining, for a point or multiple points in the first image, a matching location or multiple matching locations in a second image. The first image and the second image can be of a patient. Determining the matching location can involve using a rotation and scale invariant geometrical relationship. The matching locations can be used as the basis for the overlay.

FIELD OF THE INVENTION

The invention relates generally to the field of computer assistedsurgeries. In particular, the invention relates to image guidedsurgeries.

BACKGROUND OF THE INVENTION

Currently, computing systems for assisted surgeries exist (e.g., imageguided surgical systems). Some current systems include displays that canallow persons (e.g., medical professionals, surgeons, nurses, and/orother persons) to view medical data while a medical procedure (e.g., asurgery) is performed. In some systems, a desired location for an objectwith respect to the surgery can be displayed to a surgeon. In somesystems, an object image can be superimposed on (e.g., overlaid with) anintraoperative image. For example, during intraocular lens (IOL)placement.

In the case of IOL placement, some types of IOLs can require that theIOL is positioned in a specific orientation and/or location within apatient's eye (e.g., toric IOLs, multifocal IOLs). In current systems, adesired orientation and/or location for the IOL with respect to apreoperative image of an eye (e.g., an image of an eye taken prior tothe surgery) can be determined by, for example, various currentdiagnostic devices. The preoperative image can be captured by thediagnostic device concurrently with sampling the data that is used forcalculating the desired IOL positioning (orientation and/or location).

Typically, the position/condition of an eye during preoperative imagingis at least slightly different form the position/condition of the sameeye during the surgery. For example, a patient may be sitting when thepre-operative image is taken vs. lying down during the surgery. Inanother example, the eye may have drops and/or tools inserted during thesurgery that are not present during the preoperative imaging.Differences in the position/condition of the eye at the timepreoperative imaging is done versus the time surgery is performed cancause differences between information in the imaging obtained in apreoperative imaging stage and information obtained during imagingduring the surgery (e.g., the intraoperative imaging).

Some current systems can generate a list of feature pairs, one in eachof the preoperative image and the intraoperative image, that are assumedto be identical, and use these feature pairs to calculate a globalmapping from preoperative image to the intraoperative image. In mappingthe pre-operative image onto the intraoperative image, some priorsystems use shifts and/or rotations, and find a best fit. Currentsystems that typically track and/or lock on images can also typicallyrequire an orientation calculation, a shift calculation, and/or an imagecorrelation calculation also calculating a best fit, e.g., a singlemathematical function to best fit. However, there can be distortions inthe images due to, for example, liquid inserted into the eye and on theeye, tubes in the eye, tools touching the eye, differences betweenimaging systems for the pre-operative image and the real-time image,which can cause a best fit to be erroneous.

While current systems may account for some difficulties during surgery,accuracy of the information presented, ease of use of the informationpresented, and/or timing of the information presented can be improved.Therefore, it can be desirable to improve electronically assistedsurgical systems.

SUMMARY OF THE INVENTION

In one aspect, the invention involves a method for determining, for apoint in a first image, a matching location in a second image, the firstimage and the second image are of a patient. The method involves (a)receiving, via a computing device, the point having a location in thefirst image. (b) receiving, via the computing device, two feature pairswherein each feature pair comprises one feature location from the firstimage and a matching feature location from the second image. (c)determining, via the computing device, a first triangle with respect tothe first image, such that the vertexes of the first triangle are thelocations of the two features of the two feature pairs that are from thefirst image and the location of the point in the first image. (d)determining, via the computing device, a second triangle with respect tothe second image, such that two of the vertexes of the second triangleare the locations of each of the respective matching features of the twofeature pairs that are from the second image, and such that the secondtriangle has triangle similarity with the first triangle, yielding alocation of a third vertex, and (e) determining, via the computingdevice, the matching location in the second image of the point in thefirst image based on the location of the third vertex, wherein the firstimage and the second image are selected from at least one of: apreoperative image and an intraoperative image of the patient.

In some embodiments, the two feature pairs are selected from a set offeature pairs. In some embodiments, the two feature pairs are selectedbased on distance of the two features of the two feature pairs that arefrom the first image to the point. In some embodiments, an order of thevertices in the second triangle proceeding clockwise is the same as theorder of the respective matching vertices in the first triangleproceeding clockwise.

In some embodiments, each vertex angle in the second triangle is equalto the vertex angle of its respective matching vertex in the firsttriangle. In some embodiments, the images of the patient are images ofan eye of the patient. In some embodiments, the images are of anexterior eye and the feature locations are in the limbus area of the eyein the images. In some embodiments, the images are of the retina of theeye.

In some embodiments, the method involves repeating the method of claim 1for each of multiple points, wherein the multiple points are generatedfrom guidance information that is to be overlaid on the second image andwherein the two feature pairs are updated for each of the multiplepoints.

In some embodiments, the method involves repeating steps (b) through (d)for each of a plurality of feature pairs to determine a plurality ofcorresponding third vertex locations, calculating an averaged thirdvertex location, ad determining the matching location in the secondimage of the point in the first image based on the averaged third vertexlocation.

In some embodiments, the method also involves in step (e), setting thelocation of the third vertex as the matching location in the secondimage of the point in the first image. In some embodiments, the methodalso involves setting the averaged third vertex location as the matchinglocation in the second image of the point in the first image.

In another aspect, the invention involves a method for determining, fora point in a first image, a matching location in a second image, thefirst image and the second image are of a patient. The method involvesreceiving, via a computing device, the point having a location in thefirst image. The method also involves receiving, via the computingdevice, at least two feature pairs wherein each feature pair comprisesone feature location from the first image and a matching featurelocation from the second image, wherein each of the features in eachfeature pair has a corresponding location in its respective image. Themethod also involves determining, via the computing device, ageometrical relationship with respect to the first image, based on thelocations of the at least two features pairs and the location of thepoint in the first image, wherein the geometrical relationship is scaleand rotation invariant. The method also involves determining, via thecomputing device, the matching location in the second image based on thegeometrical relationship and the locations of the respective matchingfeatures of the at least two feature pairs that are from the secondimage, wherein the first image and the second image are selected from atleast one of: a preoperative image and an intraoperative image of thepatient.

In another aspect, the invention involves a method for filtering a setof feature pairs between two images, the two images are of a patient.The method involves receiving, by a computing device, a set of featurepairs wherein each feature pair comprises one feature location from afirst image and a matching feature location from a second image. Themethod also involves filtering, by the computing device, the set offeature pairs based on at least one of a scale invariant and arotational invariant geometrical relationship between feature locationsfrom the set of feature pairs, wherein the two images are selected fromat least one of: a preoperative image and an intraoperative image of thepatient.

In some embodiments, the filtered set of feature pairs has less featurepairs then the received set of feature pairs. In some embodiments,filtering the received set of feature pairs reduces false feature pairsin the set of feature pairs.

In some embodiments, the filtering also involves determining, by thecomputing device, a set of couples of features, determining, by thecomputing device, for each couple of the set of couples a first distancebetween the two feature locations of the two feature pairs of the couplethat are from the first image, determining, by the computing device, foreach couple of the set of couples a second distance between the twofeature locations of the two feature pairs of the couple that are fromthe second image, determining, by the computing device, for each coupleof the set of couples a ratio between the first distance and the seconddistance, yielding a set of respective ratios, and discarding featurepairs based on the set of respective

In some embodiments, the method involves determining, by the computingdevice, a relative scale between the two images, based on the set ofrespective ratios, and determining, by the computing device, a selectedfeature pair to be discarded from the set of feature pairs based on thedetermined relative scale and based on the ratios that were determinedusing the selected feature pair.

In some embodiments, the filtering involve determining, by the computingdevice, a set of couples of feature pairs, determining, by the computingdevice, for each couple of the set of couples a first angle of a lineconnecting the two feature locations of the two feature pairs of thecouple that are from the first image, determining, by the computingdevice, for each couple of the set of couples a second angle of a lineconnecting the two feature locations of the two feature pairs of thecouple that are from the second image, determining, by the computingdevice, for each couple of the set of couples a difference of anglesbetween the first angle and the second angle, yielding a set ofrespective differences of angles, and discarding feature pairs based onsaid set of respective differences of angles.

In some embodiments, the discarding involves determining, by thecomputing device, a rotation between the two images, based on the set ofrespective differences of angles, and determining, by the computingdevice, a selected feature pair to be discarded from the set of featurepairs based on the determined rotation and based on the differences ofangles that were determined using the selected feature pair.

In some embodiments, the filtering involves selecting at least threefeature pairs from the set of feature pairs, determining, via thecomputing device, a first polygon with respect to the first image, suchthat the vertexes of the first polygon are the feature locations of theat least three feature pairs that are from the first image, determining,via the computing device, a second polygon with respect to the secondimage, such that the vertexes of the second polygon are the featurelocations of the at least three feature pairs that are from the secondimage, determining via the computing device, whether the first andsecond polygons are similar and when the two polygons are not similar,discarding from the set of feature pairs at least one feature pair ofthe at least three feature pairs.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of embodiments of the disclosure are describedbelow with reference to figures attached hereto that are listedfollowing this paragraph. Dimensions of features shown in the figuresare chosen for convenience and clarity of presentation and are notnecessarily shown to scale.

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features and advantages thereof, can beunderstood by reference to the following detailed description when readwith the accompanied drawings. Embodiments of the invention areillustrated by way of example and not limitation in the figures of theaccompanying drawings, in which like reference numerals indicatecorresponding, analogous or similar elements, and in which:

FIG. 1 is a flow chart showing a method for locating data from a firstimage on a second image, according to some embodiments of the invention.

FIG. 2 is diagram showing an example of a display device displayingguidance information as an overlay superimposed on an intraoperativeimage, according to some embodiments of the invention.

FIG. 3A is a diagram showing an example of a display device showing apreoperative image and an intraoperative image, according to someembodiments of the invention.

FIG. 3B is a diagram showing an example of a display device showing PIP,according to some embodiments of the invention.

FIG. 3C is a diagram showing an example of the desired and current IOLalignment displayed without the preoperative image, according to someembodiments of the invention.

FIGS. 4A-4C are images showing an example of an OCT B-scan located on anintraoperative image, according to an illustrative embodiment of theinvention.

FIG. 4D is an example of displaying an overlay on a still image of anintraoperative image, according to an illustrative embodiment of theinvention.

FIG. 5 is flow chart showing a method for determining, for a point in afirst image, a matching location in a second image, according to someembodiments of the invention.

FIG. 6A is a diagram showing an example of a point in a first imagebeing copied to a second image using the method of FIG. 5, as describedabove, according to some embodiments of the invention.

FIG. 6B shows images of an example of copying a location between twoactual images of an eye, according to illustrative embodiments of theinvention.

FIG. 7 is a flow chart showing a method for determining a set of featurepairs between features of two images, according to some embodiments ofthe invention.

FIG. 8A shows a geometrical relationship being used to filter a set offeature pairs, according to some embodiments of the invention.

FIG. 8B are diagrams that illustrate two histograms of ratios generatedfrom actual images of a retina, according to some embodiments of theinvention.

FIG. 9 is a diagram of an example of a system for overlaying guidanceinformation, according to some embodiments of the invention.

FIG. 10 shows a block diagram of a computing device 1400 which can beused with embodiments of the invention.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the invention.However, it will be understood by those skilled in the art that theinvention can be practiced without these specific details. In otherinstances, well-known methods, procedures, and components, modules,units and/or circuits have not been described in detail so as not toobscure the invention.

Generally, the invention can involve displaying an image of a medicalprocedure (e.g., an intraoperative image) with additional information(e.g., data) that can augment the image of the medical procedure.

The data can be overlay data. The overlay data can be shapes,information (e.g., guidance information), or any visual output that isdesired to be displayed concurrent with or appear on top of theintraoperative image. The overlay data can be images that appear to bepositioned on top of the intraoperative image. The overlay data can bedisplayed such that it varies in color, intensity, opacity,transparency, or any combination thereof.

Each pixel on the image to be displayed during the medical procedure canbe assigned a value. The value for each pixel can depend on theintraoperative image and the overlay data. The overlay data can causethe pixels in the intraoperative image to be modified, according to thedesired image output for the medical procedure.

Overlaying data with an image (e.g. a preoperative image or anintraoperative image) can assist a medical professional during a medicalprocedure. For example, during ophthalmic surgery it can be desirablefor a surgeon to understand where surgical incisions or treatments areplanned to take place, and/or where medical objects are planned to beplaced, such as for example for capsulotomy, limbal relaxing incisions(LRIs), stitches, retinal laser photocoagulation, and/or IOL insertion.The planning information can be presented to the surgeon as overlay dataon the intraoperative image. For example, when inserting a toric IOLinto an eye, the correct orientation of the IOL can be imperative to thepatient's ability to see properly after the surgery. Toric IOLstypically include markers (e.g., axis marks) that can help a surgeonalign the IOL within the patient's eye properly. However, the IOL'smarkers can only assist in the IOL's placement within the patient's eyeif there is an accurate, stable, and timely reference.

The invention can include methods and systems that can allow anaccurate, stable and/or timely reference for IOL alignment. As isapparent to one of ordinary skill in the art, the method and systemsdiscussed herein can be applicable to any electronic assisted surgerywhere an object (e.g. a tool, an implant and/or any medical object) ispositioned at a predefined desired location and/or orientation on orwithin a patient, or where a surgical treatment (e.g. an incision, abiopsy, and/or a tumor ablation) is taken place at a preplannedlocation.

In general, the invention can allow for i) overlaying preoperativeguidance information (e.g., guidance data) with an intraoperative image,e.g., overlaying information from a pre-operative image, or informationassociated with a preoperative image, with an intraoperative image thatcan be a live image or a still image taken at a particular moment intime of the real-time surgical procedure, ii) overlaying an indicator ofa current location and/or orientation of a medical object during themedical procedure with a preoperative image, e.g. a preoperative imageas taken by a diagnostic device, or a preoperative image alreadyoverlaid with preplanning or guidance information, and iii) overlayingintraoperative guidance information with an intraoperative image, e.g.overlaying information associated with a live image viewed by onesurgeon with a live image viewed by another surgeon, or overlayinginformation associated with a still image taken at a particular momentduring the surgical procedure with a live image. As an example foroverlaying preoperative guidance information with a live image (e.g., anintraoperative image), a surgeon may view a guidance information overlayindicating a preplanned location and/or orientation of an IOL overlaidwith the live image while moving and/or rotating the IOL. As an examplefor overlaying intraoperative guidance information with the live image,a supervising surgeon may draw a line indicating a desired location foran incision on an intraoperative image (e.g. a snapshot of the liveimage), and a resident performing the procedure may view the lineoverlaid with the live image. As an example for overlaying guidanceinformation with a preoperative image, a surgeon may view thepreoperative image overlaid with guidance information concurrently withviewing the live image (e.g. in a picture-in-picture view or aside-by-side view).

In the latter example, two indicators (e.g. two lines for indicatingorientation and optionally also location, or two circles for indicatinglocation) may be overlaid with the preoperative image. A first indicatormay represent the desired location and/or orientation of an IOL, and canremain unchanged with respect to the preoperative image it is overlaidupon. However, a second indicator may represent the actual IOL locationand/or orientation during the medical procedure, and can be continuallyupdated (e.g., with a predefined periodic rate) as the surgeonmanipulates the IOL. The surgeon can see in real-time an indicatorrepresenting the actual IOL location and/or orientation with respect toan indicator representing the desired IOL location and/or orientation,and can change the IOL location and/or orientation until the two symbolsare in alignment (e.g., the surgeon can concurrently see both theguidance overlay on the preoperative image, and the IOL in the liveimage).

In some embodiments, a surgeon may be concerned with only the IOLorientation and not the location. In these embodiments, for example whenthe two indicators (e.g. lines) are parallel to each other they can beconsidered to be in alignment. In some embodiments, a surgeon isconcerned with only the location of the medical device. In theseembodiments, for example, when the two indicators (e.g. circles) areconcentric they can be considered to be in alignment. In someembodiments, a surgeon is concerned with both the location and theorientation. In various embodiments, the surgeon can decide when the twoindicators have reached a sufficiently aligned state.

Generally, the methods and systems can allow for i) displaying with theintraoperative image indicators that are associated with locations in apreoperative image, ii) displaying with the preoperative imageindicators that are associated with locations in the intraoperativeimage, and iii) displaying with one intraoperative image indicators thatare associated with locations in another intraoperative image.

Accurately matching (e.g., locating) a point in one image (e.g., a firstimage) on another image (e.g., a second image) can involve finding amost accurate location on the second image that corresponds to the pointon the first image. Accurately matching a point in one image on anotherimage can also be referred to as copying the point from one image toanother image. Although the two images may be of the same object (e.g.,two images are of the same patient's eye), the two images may not to beidentical (e.g., the two images are of the same patient's eye taken attwo different times). Thus, matching a point (e.g., a desired locationfor a center of an IOL) in a first image (e.g., preoperative image) to apoint in a second image (e.g., a intraoperative image) can involvelocating a point on the second image that closely corresponds to thepoint on the first image, as described in further detail below withrespect to FIG. 1

In some embodiments, locating the correct point can involve finding alocation relative to adjacent anatomical elements in both images thatare identical. The anatomical elements of a reference can depend on thetype of the surgical procedure and/or the stage of the procedure. Forexample, in open brain surgery, superficial cortical vasculature canserve as reliable anatomical elements for transfer of locations at theinitial stages of the procedure, but once a tumor is partially removedthey may be considered unreliable as the brain tissue is subject todeformation. In another example, in anterior ophthalmic proceduresanatomical elements near the limbus may be considered reliable, andconjunctival blood vessels distant from the limbus may be consideredunreliable. This can be due to possible movement of the conjunctivarelative to the sclera due to surgical manipulations.

Locations (e.g., one or more points) in each of the respective imagescan be transferred to the other images by determining correspondinglocations in the other respective image. In some embodiments,corresponding locations can be determined based on aligning the twoimages. In some embodiments, corresponding locations can be determinedas described in further detail in FIG. 1 below. A location can be an XYcoordinate system location in the image, having sub-pixel resolution(e.g., non-integer x and y values). Each image can be defined as havingpixels, pixel locations, and the pixel locations can be defined in an XYcoordinate system. The location of points and/or objects within imagescan also be referred to in an XY coordinate system.

FIG. 1 is a flow chart showing a method 100 for locating data from afirst image on a second image, according to some embodiments of theinvention. The method can include receiving the first image (Step 110).

The first image (e.g., reference image) can be a preoperative image. Thepreoperative image can be an image of a patient taken prior to anoperation. The first image can show a region of interest which can be aportion of patent that is of interest, for example, for a procedure. Forexample, an eye, a brain, a lung or a blood vessel.

The first image can be a two-dimensional (2D) image, for example inophthalmic surgery an image of an eye generated by a CCD- or CMOS-basedcamera, or an image of a retina generated by an SLO camera. The 2D imagecan be generated from a three-dimensional (3D) imaging dataset, forexample in brain surgery, an oblique slice generated from a CT imagingdataset, or a rendered image of a 3D segmented model of a tissuegenerated from an MRI imaging dataset, or for example in ophthalmicsurgery an en-face image or a summed voxel projection image generatedfrom an OCT imaging dataset.

The method can also include receiving data (e.g., overlay data) that isassociated with the first image that can be represented as an overlaythat is displayed on a second image (e.g., an intraoperative image)(Step 120). The overlay data can be guidance information for the medicalprocedure determined during the capturing of the preoperative image. Theguidance information (e.g., reference data) can include a desiredlocation and/or orientation for a medical object within the first image.The guidance information can be a desired location for insertion,placement and/or positioning of any kind of the medical object withrespect to the patient. For example, the data can be xy coordinates of acenter of a visual axis and an orientation, and it can be represented asa line centered and oriented per the data and is displayed as an overlaywith the second image. The guidance information can be a desiredlocation for a surgical treatment. For example, the data can be acontour (e.g., an arc or a line) of a planned incision, a location of aplanned biopsy, and/or a location of a planned tumor ablation, that canbe represented as overlays on the second image.

The guidance information can indicate a portion of an eye. The guidanceinformation can indicate a particular object in the first image. Theguidance information can have characteristics that can cause it to bedisplayed as a line, a dot, a series of dots, a color, or any visualindicator as is known in the art. In some embodiments, for an IOLplacement, the guidance information can indicate an orientation relativeto the image for IOL placement and/or the desired IOL placementlocation.

In some embodiments, the guidance information is received by the system.In some embodiments, the guidance information is automatically generatedby a diagnostic device. In some embodiments, the guidance information isnot automatically generated by a diagnostic device. In variousembodiments, if the guidance information is not automatically generatedby a diagnostic device (e.g., such as the toric IOL location andorientation data), or if a surgeon chooses to change automaticallygenerated guidance information, the surgeon may use a SW tool (e.g. adedicated SW tool) for overlaying or drawing on the preoperative image(or a first image) any of the following: circles, lines, othergeometrical shapes, freehand drawing. The overlays can indicate areas byshapes filled. e.g. by texture or color. The SW tool may store theoverlay as textual data that is used during a procedure to generate anoverlay on the intraoperative image. For example, guidance informationfor a capsulotomy can be a circle having a known diameter drawn as anoverlay with the preoperative image.

The method can also involve determining one or more points based on thereceived data (Step 125). The received data (e.g. guidance information)can be represented as a series of locations (e.g. points) in the firstimage coordinate system (CS). For example, a circle can be representedby 10 points uniformly spread along the circle. In these embodiments,during a procedure, the points can be located in the intraoperativeimage (e.g., a live image). Locating the points in the intraoperativeimage can involve, copying the points from a preoperative CS to anintraoperative CS. In some embodiments, once the points are copied tothe intraoperative image CS, the method can involve reconstructing theguidance information based on the copied points. For example,calculating a circle that fits (e.g., best fits) the 10 copiedlocations. In these embodiments, the reconstructed guidance informationcan be displayed as an overlay with the intraoperative image.

In some embodiments, the received data is represented as points prior tothe procedure. In some embodiments, the received data is represented aspoints prior during the procedure. For example, the circle that thesurgeon draws on the preoperative image is guidance information that isassociated with the preoperative image (e.g., and/or stored) as, forexample, a center (x and y values) and radius (in pixels), and thepoints to be copied can be generated during a procedure that takes placeafter the surgeon draws on the preoperative image (e.g., generated inreal-time during the surgery).

In some embodiments, limbal relaxing incisions (LRIs) are represented byguidance information showing short line segments or shorts arcs having ageometrical representation in the preoperative image CS, or as a seriesof points. In some embodiments, areas where a retina is detached areindicated on a preoperative image of a retina by a contour filled with acolor.

In various embodiments, guidance information that is displayed as anoverlay can be represented for instance as an OpenGL object (e.g. by alist of vertexes and textures) that can include a finite number oflocations that may be copied from one image CS to another (e.g., apreoperative image to an intraoperative image). In various embodiments,the first image can be an intraoperative image (e.g., a live image) andthe guidance information can be automatically generated by detecting IOLaxis marks. In various embodiments, the guidance information isautomatically and/or manually generated, in association with either apreoperative image or an intraoperative image. In various embodiments,the guidance information is automatically generated with respect to apreoperative image (e.g. by a diagnostic device), manually generatedwith respect to a preoperative image (e.g. by a surgeon as describedabove), automatically generated with respect to an intraoperative image(e.g. by detecting or tracking a tool or an implant with respect to thelive image), and/or manually generated with respect to an intraoperativeimage (e.g. by drawing a teaching symbol on an intraoperative snapshot).

In various embodiments, the first image is an intraoperative image. Theintraoperative image can be a real-time image. The intraoperative imagecan be a live image or a still image. As is apparent to one of ordinaryskill in the art, the live image can be a series of still imagescontinuously changing in real-time.

In various embodiments, the second image is an intraoperative image.

In various embodiments, the first image is a still image (e.g., asnapshot) of the intraoperative image and the second image is a liveimage. In some embodiments, the guidance information can be a line drawnon the snapshot, for example, by a senior surgeon.

In various embodiments, the first image is a preoperative image. Thepreoperative image can be a still image or a video image. In variousembodiments, the second image is a preoperative image.

In various embodiments, the first image is a preoperative image and thesecond image is an intraoperative image (e.g., a live image orsnapshot). In these embodiments, the guidance information can begenerated by a diagnostic device or by the surgeon, and can berepresented as locations in the preoperative image CS

The method can also include receiving the second image (Step 130). Forexample, the second image can be received during the medical procedure.The second image can be an intraoperative image (e.g., a live image ofthe medical procedure or snapshot of the medical procedure), apreoperative image and/or any image as described with respect step 120.

The method can also include, matching locations of one or more pointsfrom the first image to the second image (Step 140). The one or morepoints can be based on the received data, for example, as describedabove in Step 125. For example, the guidance information associated withthe preoperative image can be represented by points (e.g., two pointsrepresenting a line) in the preoperative image CS. The method can alsoinvolve displaying guidance information as overlaid with the secondimage based on the one or more points (Step 150).

In some embodiments, when the guidance information is an indicator thatis overlaid on a live image, it is locked to anatomical elements ofreference in the live image such that the guidance information moveswhen there is movement in the live image.

In some embodiments, when the guidance information is an indicator thatis associated with a location of an object in the live image and it isoverlaid on a preoperative image, the overlay can dynamically changewhen the object moves relative to anatomical elements of reference inthe live image.

In some embodiments, multiple preoperative diagnostic tests are averagedto generate the guidance information. For example, in toric alignmentguidance, when multiple astigmatism measurements are preoperativelyperformed by one or more devices, the various results are accuratelyaveraged when, for instance, each measurement is accompanied with animage of the eye sampled concurrently with the diagnostic test. Invarious embodiments, the method involves selecting one of the multiplediagnostic images as a reference image (e.g., first image), andconverting the various astigmatism measurements from the CS of each ofthe multiple diagnostic images to the CS of the reference image. Thiscan be based on aligning each of the diagnostic images to the selectedreference image, and adjusting the corresponding astigmatismmeasurements according to the relative rotation between the images. Insome embodiments, an astigmatism measurement can be represented as twopoints in the diagnostic image CS, and the location of these two pointsis copied to the reference image CS. In some embodiments, averagingmultiple preoperative diagnostic tests to generate the reference datainvolves automatically discarding outlier results. In some embodiments,the user decides which results to include in the averaging.

Turning to FIG. 2. FIG. 2 is diagram showing an example of displaydevice 200 displaying guidance information as an overlay in the form ofa dashed line 210 (e.g., desired IOL location and orientation)superimposed on the intraoperative image 220 showing IOL 230 after beinginserted to an eye 250, according to some embodiments of the invention.The current orientation of the IOL is apparent via IOL markers 260.

The toric IOL in FIG. 2 includes six dots (or markings) that canindicate a steep or flat axis of the toric IOL. Toric IOLs typicallyinclude axis marks, such as a series of dots, lines, rectangles and/orother markings as are known in the art, that can assist a surgeon toalign the IOL to a predetermined orientation. These markings can bedetected in an intraoperative image, e.g. by a template-matchingalgorithm, a deep-learning algorithm, and/or any algorithm known in theart to detect the markings. Once the locations of the IOL markings inthe intraoperative image are known, the system can determine, forexample, the location of the center of the IOL in the intraoperativeimage. The guidance information (e.g., overlay) can correspond to theoptical axis that the axis marks indicate, e.g., an overlay for an IOLwith indications for a steep axis that is orthogonal to an overlay foran IOL with indications for a flat axis (e.g., either the IOL type canbe detected automatically or a user can indicate if the IOL type isdifferent from a default type).

In various embodiments, for IOL insertion, the guidance informationindicates a desired location and/or orientation of the IOL. In someembodiments, the guidance information is locked (e.g., pinned) to adesired location (e.g. relative to the eye). In some embodiments, fortoric IOL insertion, the guidance information indicates the desiredorientation of the IOL.

In some embodiments, for toric IOL insertion, the guidance informationis displayed as an overlay centered on a center of the IOL as detectedin the intraoperative image. In this manner, the indicator of thedesired IOL orientation can be pinned (or locked) to the IOL center. Forinstance, the user may choose to pin the overlay to the IOL center oncethe IOL is at its final position, e.g., for fine tuning and/or forverifying the alignment. When a user chooses to pin the overlay to theIOL center, and the IOL center is temporarily not being detected (e.g.due to obscurations by the iris or by tools), the overlay can becentered on either one of the last determined IOL center location or thepreoperatively determined (e.g., desired) IOL location.

In some embodiments, when the intended overlay location changesabruptly, the overlay is gradually shifted to the updated (e.g.,intended) location. Abrupt changes in the intended overlay location canoccur when a user changes the selection of overlay centering frompinning to the preplanned (e.g., desired) location to pinning to the IOLcenter, or vice versa, or when the overlay is chosen to be pinned to theIOL center but the IOL center is temporarily not detected.

Detecting the IOL center may be based, for example, on detecting the IOLcontour or detecting the IOL axis marks. When the IOL shape is known,detecting the IOL center can be based on detecting segments of the IOLboundaries. For example, when the IOL is round (e.g., disregarding thehaptics), segments of the IOL edges can be detected, and/or the centerof a circle that best fits these segments can be determined as the IOLcenter. Detecting the axis marks locations may be determined for examplebased on the IOL type (e.g., a known IOL type has a known number of theaxis marks and known distances between the axis marks). In anotherexample, the axis mark locations may be determined based on estimatingthe most likely location based on a nominal IOL diameter and/or based onknown patterns for various toric IOLs in the market.

In some embodiments, the angular difference between the desired IOLorientation and the current IOL orientation is determined. The angulardifference can be conveyed to a user various visual and/or auditoryways.

In some embodiments, the angular difference between a desired IOLorientation and the current IOL orientation can be determined, forexample, by detecting the IOL axis marks in the intraoperative image,and copying the axis marks locations from the intraoperative image CS tothe preoperative image CS.

An angle of a line that best fits the copied locations in thepreoperative image CS may be calculated, and the difference between thisangle and the preplanned desired angle may be determined.

In various embodiments, the method can involve generating and/orreporting a confidence score that can control an appearance of theguidance overlay. For example, if a surgeon is causing rapid movement inan eye, the color of the guidance overlay can change to reflect that itmay be less accurate during the rapid movement. The confidence score canalso be conveyed to the user via a separate symbol (e.g., a coloredand/or numbered meter).

In some embodiments, the system may use auditory indications to assistthe surgeon, e.g. when the surgeon chooses not to see an overlay on alive image. For example, the system can be programmed to calculate thepositional and/or angular difference (e.g., the measure of alignment)between a desired IOL location and/or orientation and a current IOLlocation and/or orientation. The measure of alignment can be determinedwithout actually displaying the indicators that represent the desiredIOL location and/or orientation and the current IOL location and/ororientation, and can be calculated based on their relative locationand/or orientation. The calculations can be based on copying thelocations of the detected axis marks from the intraoperative image CS tothe preoperative image CS, and/or by copying the desired IOL locationand/or orientation from the preoperative image CS to the intraoperativeimage CS. When the IOL does not have axis marks and the guidanceinformation includes a desired location only, the IOL location can bedetected in the live image for instance based on detecting its contourin the live image. Having calculated a measure (or score) of alignment,the system can generate an auditory indicator that indicates to thesurgeon how close the actual IOL location and/or orientation is withrespect to the desired IOL location and/or orientation. Such anindicator may be for instance a beeping sound having a frequencyincreasing with the alignment score.

In some embodiments, the surgeon relies on an auditory indicator alone.In some embodiments, once an alignment score threshold is met, a PIP canbe presented to the surgeon automatically. In some embodiments, the PIPcan appear when the surgeon presses a footswitch. In some embodiments, asurgeon momentarily freezes a live image to see the desired indicatoroverlaid with the intraoperative snapshot image, before returning to thelive image to improve the alignment.

In some embodiments, known models of particular IOLs are used. In someembodiments for other medical procedures, models of known medicalobjects/devices can be used for detection in the intraoperative image.

FIG. 3A is a diagram showing an example of a display device 300 showinga preoperative image 320 and an intraoperative image 340, according tosome embodiments of the invention. As shown in FIG. 3A, the preoperativeimage 320 shows a desired IOL location and orientation 324 and a currentIOL location and orientation 326. The intraoperative image includes IOLsaxis marks (or markers) 310 indicating an axis of the IOL. The axismarks may be detected (e.g. by image processing algorithms) in theintraoperative image, and their locations can be copied to thepreoperative CS e.g., via the algorithm shown below in FIG. 5, and beused to generate the current IOL location 326.

In some embodiments, the preoperative image and the intraoperative imageare concurrently displayed in a side-by-side layout, as in FIG. 3A. Insome embodiments, for a medical procedure involving an IOL, thepreoperative image including a desired and a current IOL alignment andthe intraoperative image are concurrently displayed as a“Picture-in-Picture” (PIP). FIG. 3B is a diagram showing an example of adisplay device 300 showing PIP, according to some embodiments of theinvention. The PIP includes the preoperative image 320 showing a desiredIOL location and orientation 324 and current IOL location andorientation 326 (e.g., guidance information) that is the IOL locationand orientation in a current medical procedure. In FIG. 3B, theintraoperative image 340 shows an IOL 301. The PIP shows a desired IOLlocation and orientation 324 and current IOL location and orientation326 (e.g., guidance information) that is the IOL location andorientation in a current medical procedure. In various embodiments, thepreoperative image 320 is positioned at other desired locations withinthe display device. In some embodiments, the preoperative image 320 ispositioned such that the preoperative image and the intraoperative imagecan be viewed simultaneously. In some embodiments, the intraoperativeimage 340 is positioned within the preoperative image 320.

In various embodiments, the user can provide an input for positioningthe preoperative and intraoperative images relative to each other,selecting the size, magnification, centering and/or othercharacteristics of each image individually to, for example, allow theuser to arrange the images to have an unobstructed view of the relevantpal of the patient site of interest. In various embodiments, thepreoperative image is displayed within the boundaries defining the areaof the intraoperative image, outside of the boundaries defining the areaof the intraoperative image or any combination thereof. As is apparentto one of ordinary skill in the art, the preoperative image can bepositioned/placed in other locations within the display device.

In some embodiments, the indicators for the desired and current IOLalignment are displayed without the preoperative image, as the twoindicators can be sufficient for positioning the IOL (e.g., a surgeonmay move and/or rotate the IOL until two indicators are aligned,regardless of the background image). For example, turning to FIG. 3C.FIG. 3C is a diagram showing an example of the desired and current IOLalignment displayed without the preoperative image, according to someembodiments of the invention. In some embodiments, the desired andcurrent IOL alignment are displayed are superimposed upon a schematicimage (e.g., synthetic representations of an eye, preoperative pupil,center of the pupil). In some embodiments, the first image (e.g., thereference image) is an image of a retina generated by a diagnostic OCTdevice, accompanied by information regarding the locations, relative tothe image of the retina, of OCT B-scans provided by the OCT device. Suchlocation information can be provided as pairs of locations of edges oflines, each line representing an OCT B-scan location with respect to theimage of the retina. FIGS. 4A-4C are images showing an example of an OCTB-scan located on an intraoperative image, according to an illustrativeembodiment of the invention.

FIGS. 4A and 4B show images generated by an OCT imaging device takenprior to medical procedure (e.g., preoperatively), according to someembodiments of the invention. Image 400 is an image of a retinagenerated by the OCT imaging device, and image 410 is one of multipleB-scan images generated by the same device. The multiple horizontallines overlaid on the preoperative image 400 indicate locations on theretina corresponding to each of the multiple B-scans. In someembodiments, the system stores the image of the retina without theoverlaid lines, and the location of the lines with respect to the imageare stored separately (e.g. as pairs of points indicating the two edgesof each line). Line 420 can indicate the location on the retinacorresponding to B-scan 410. FIG. 4C shows an intraoperative image 450of a retina. Preoperative OCT B-scan image 410 is displayed in PIP 440.Using preoperative image 400 of the retina as a reference image (e.g.,the image without the overlaid lines is used as reference), the locationof line 420 can be copied from the preoperative image 400 to the liveimage 450, e.g., by copying the location of the two edges of the linefrom the reference image to the intraoperative image, to generateoverlay 430, which can indicate the location on the retina correspondingto the OCT B-scan image 410 displayed in PIP 440. In some embodiments,the user scrolls within the set of B-scan images, and as the OCT B-scanimage displayed in PIP 440 changes, the location of the correspondingline on the intraoperative image is determined and displayed as anoverlaying line. The determination for the corresponding location on theintraoperative image can be performed on a frame-by-frame basis, evenwhen the preoperative OCT image displayed in PIP 440 has not beenchanged by the surgeon, as image 450 can be real-time and the overlaycan be dynamically updating.

In some embodiments, the preoperative image is a diagnostic image of theretina on which the surgeon made pre-planning notes and markings, andthe intraoperative image is a live image of the retina. Pre-planningmarkings can be for example a location of a membrane that is to beremoved during the surgery, and/or a location where a retinal detachmentappears. In some embodiments, it can be desirable for a surgeon to viewthe markings overlaid with the live (e.g., intraoperative) image, e.g.,copied or located from the preoperative image to the live image. Copyinga marking may be performed by breaking the marking into smallline-segments, and copying each of the line edges from the referenceimage (e.g., the preoperative image) to the target image (e.g., the liveimage). Each line edge can have a defined location in the image. Ingeneral, any shape and color may be represented by locations, e.g. byvertexes, for instance as in an OBJ file. Typically, the overlay isstored separately from the image (e.g., not saved directly on thereference image), using the same CS as the reference image CS, such thatthe overlay may be copied from the reference image CS to the targetimage CS.

In some embodiments, one or more still images of a live image iscaptured by the user, and the overlay appears only on the one or morestill images. For example, for procedures where an overlay indicates anarea of a detached retina (e.g., a posterior segment ophthalmicprocedure) a surgeon may prefer that the overlay does not appear all thetime, rather that it appears only responsive to a user-based inputreceived by the display system (e.g. by pressing the footswitch). Insome embodiments, this can be preferred since the overlay in someembodiments, can obscure the area that the surgeon needs to attend. Insome embodiments, when the image is still (e.g., frozen), the system canadd a clear indication or warning so the surgeon is aware that the imageis frozen. In some embodiments, the overlay only appears on the liveimage during a continuous user-based input (e.g. only while the user ispressing down on a footswitch). FIG. 4D is an example of displaying anoverlay on a still image of an intraoperative image 460 (e.g. when thelive image is frozen), according to an illustrative embodiment of theinvention. The overlay 480 indicates an area of a detached retina duringa posterior segment ophthalmic procedure. e.g. indicated by coloring anarea on an intraoperative image of the retina that is detached, andindicating by small crosses 470 locations for laser photocoagulation.Indication 490 is a warning to alert a surgeon that the image is frozen.In some embodiments, displaying the live image frozen with the overlayis time limited, for instance it is shown only when the surgeon pressesthe footswitch.

In some embodiments, overlaying guidance information on a still imagecan provide an advantage of avoiding jitter that can arise due to, forexample, micro-saccades of the patient's eye and/or due to unstableoverlay location on the live image (e.g. instability that is affected bysmall changes in the live image). For example, overlaying the guidanceinformation on a snapshot can allow for improved verification of analignment between an IOL and a guidance overlay.

In some embodiments, a still image of a live image is displayed in aside screen or in PIP, and the still image includes the overlay. Thiscan allow the surgeon to see both the live image (e.g., without anyobscurations that may interfere with what the surgeon is doing) and theguidance information (overlaid on the snapshot). In some embodiments,two representations of the live image are displayed to the surgeon, onewithout an overlay and one with an overlay. For instance the live imagewith the overlay is displayed in a side screen or in PIP. This can allowthe surgeon to see both the live image without any obscurations that mayinterfere with what the surgeon is doing and the live image with theguidance information(overlaid on the live image in PIP.

In some embodiments, for instance in a system for brain surgery, thereference image (e.g., the first image) is rendered from a CT or MRIdataset, from the perspective of the camera that is generating theintraoperative image. In some embodiments, when the user views astereoscopic (3D) intraoperative image generated by two cameras, aseparate reference image is used for copying information to each of thetwo intraoperative (e.g. live) images. For example, two images can berendered from a CT or MRI dataset from two perspectives, correspondingto the two perspectives of the cameras generating the live image, orcorresponding to the two perspectives of the two optical channels of asurgical microscope, when the user is viewing the patient site ofinterest via the microscope and not via a display device (or devices).In another example the reference image is a scanning laserophthalmoscopy (SLO) image generated for instance by a diagnostic OCTdevice. In yet another example ophthalmic OCT data may be used togenerate an en-face image that may be used as a reference image. Inanother example a reference image may also be generated by an infraredcamera, as opposed to imaging in the visible-range. As described above,guidance information, reference data, or any information can be copied(e.g., transferred or located) from a rust image to a second image (orvice versa). The guidance information can be associated with a point inthe first image (e.g., a desired IOL marker location), or multiplepoints in the first image (e.g. a contour of a membrane on the retinarepresented by a finite number of points). Each point can be copied fromthe first image to the second image, i.e. a matching location in thesecond image can be determined for each of the multiple pointsrepresenting the guidance information.

In some embodiments, a user (e.g., a nurse or a surgeon) verifies thatimage-to-image location matching is working correctly by moving a cursor(e.g., guidance information) over the first image (e.g. preoperativeimage), as displayed for instance on a touchscreen of the systemside-by-side with the second image (e.g. intraoperative image), to pointat prominent image elements, and visually verifying that a correspondingcursor, copied from the first image to the second image, is correctlypositioned (e.g., the user may verify that the second cursor points atthe same image element in the second image). In these embodiments, thelocation matching can be performed as shown above in FIG. 5 and/or viaregistration.

In some embodiments, when image-to-image location matching is based onregistering the first image and the second image, a user (e.g., a nurseor a surgeon) verifies that the registration was performed correctly byviewing a toggled view of the first image and second image (e.g. onlyone of the two images is displayed at any given moment). In theseembodiments, the images are displayed such that if one of the imageswere partially transparent, corresponding anatomical elements in the twoimages would overlap (e.g. if the registration was performed correctly).In these embodiments, the user views either the first image after it isregistered (e.g. aligned) to the second image, alternately with thesecond image, or the second image after it is registered (e.g. aligned)to the first image, alternately with the first image. In someembodiments, the user may adjust the toggling frequency. In someembodiments, the user may adjust the registration while viewing thetoggled view of the two images. For instance, the user may rotate theregistered image such that it is better aligned with other image.

FIG. 5 is flow chart showing a method for determining, for a point in afirst image, a matching location in a second image, according to someembodiments of the invention.

The method can involve receiving, via a computing device, the pointhaving a location in the first image (Step 510). The point can have adefined pixel location (x,y) where x and y can be integer or non-integervalues.

The method can involve receiving, via the computing device, two featurepairs wherein each feature pair comprises one feature from the firstimage and a matching feature from the second image, wherein each of thefeatures in each feature pair has a corresponding location on itsrespective image (Step 520). The two feature pairs can be two featurespairs from a set of feature pairs. The set of feature pairs can be basedon matching features between the first image and the second image.Features can be matched based on similarity between descriptors. In someembodiments, even if features have sufficient similarity betweendescriptors, they may not actually be a match. In some embodiments, thefeature pairs are filtered. The filtering can be based on geometricalrelationships, for example, as described in further detail below withrespect to FIG. 7 and FIG. 8.

In some embodiments, the two feature pairs are selected from a set offeature pairs, each feature pair includes one feature from a first imageand a matching feature from the second image. The two feature pairs canbe selected based on distance (e.g., distance measured in pixels) of thetwo features of the two feature pairs that are from the first image tothe point. The distance can be normalized to the size of the anatomy asit appears in the image (e.g., the white-to-white diameter of the eye inthe image). The distance can be based on a minimum distance or a maximumdistance as input to the system (e.g., by a user, from a file or as aprogrammable parameter).

For example, assume two feature pairs, FP1a, FP1b, where FP1a haslocation (X1FP1a,Y1 FP1a) in the first image and FP1b, has location(X2FP1b,Y2 FP1b) in the second image and FP2a, FP2b, where FP2a haslocation (X1FP2a,Y1FP2a) in the first image and FP2b, has location(X2FP2b,Y2 FP2b) in the second image.

The method can involve determining, via the computing device, a firsttriangle with respect to the first image, such that the vertexes of thefirst triangle are the locations of the two features of the two featurepairs that are from the first image and the location of the point in thefirst image (Step 530). Continuing with the above example, the vertexesof the first triangle are (X1FP1a,Y1 FP1a), (X1FP2a,Y1FP2a), and (x,y),the location of the point.

The method can involve determining, via the computing device, a secondtriangle with respect to the second image, such that two of the vertexesof the second triangle are the locations of each of the respectivematching feature of the two feature pairs that are from the secondimage, and such that the second triangle has triangle similarity withthe first triangle yielding a third vertex (Step 540). Continuing withthe above example, the two vertexes of the second triangle are(X2FP1b,Y2 FP1b), (X2FP2b,Y2 FP2Ab). With the three vertexes of thefirst triangle known, angles of the first triangle, the lengths of eachside of the first triangle and/or other geometrical characteristics ofthe first triangle can be determined.

With the known angles and/or known lengths of each side and/or otherknown geometrical characteristics for the first triangle, the thirdvertex of the second triangle can be determined by finding a similartriangle to the first triangle. In some embodiments, the second trianglecan be constructed (e.g., virtually) by connecting the two matchingfeatures and drawing (e.g., virtually) two lines (e.g., rays)originating in each of the second triangle features and based on theknown angles of the first triangle that correspond to each of the twofeatures. The crossing point of the two lines can determine the thirdvertex of the second triangle and can define the third vertex. Invarious embodiments, determining the second triangle that has similaritywith the first triangle is done, as is known in the art.

In some embodiments, an order of the vertices in the second triangleproceeding clockwise is the same as the order of their respectivematching vertices in the first triangle proceeding clockwise. In someembodiments, the second triangle is not flipped with respect the firsttriangle such that the second triangle is not a mirror image of thefirst triangle. In some embodiments, each vertex angle in the secondtriangle is equal to the vertex angle of its respective matching vertexin the first triangle. For example, assume feature pair (F1, F2), whereF1 is a vertex in the first triangle and F2 is a vertex in the secondtriangle. Assume also that F1 is a vertex having an associated angle of30 degrees. Then F2 is a vertex in the second triangle that also has anassociated angle of 30 degrees. In this manner, the angles associatedwith the vertices in the first triangle are preserved for thecorresponding vertices in the second triangle. The method can involvedetermining, via the computing device, the matching location in thesecond image of the point in the first image based on the location ofthe third vertex (Step 550). In some embodiments, the matching locationin the second image can be set to the location of third vertex. As isapparent to one of ordinary skill in the art, a triangle does notactually need to be drawn, it can be virtually constructed and theproperties (e.g., angles, lengths, and/or any triangle properties and/orcharacteristics) can be stored within the computing device.

In some embodiments, the method can involve repeating the determinationof the point to be located using multiple triangles and determining thefinal result by averaging the multiple results.

As is apparent to one of ordinary skill in the art, the discussion withrespect to FIG. 5 is with respect to one point, however, in variousembodiments, the method steps of FIG. 5 can be applied to multiplepoints. For example, in the case of a line indicating a desiredorientation of an IOL with respect to a preoperative image, two or morepoints along the line (e.g., the two edges of the line) can be selectedto represent the line, and each of these points may be copied to (orlocated in) the CS of the intraoperative image. For each copied point,the feature pairs can be updated (e.g. different feature pairs are usedto copy each point). Thereafter, a line may be generated in the CS ofthe intraoperative image, e.g. by connecting the copied points.

FIG. 6A is a diagram showing an example of a point in a first image(e.g., a preoperative image, intraoperative image or a reference image)being copied to a second image (e.g., a preoperative image, anintraoperative image or a target image) using the method of FIG. 5, asdescribed above, according to some embodiments of the invention. Asshown in FIG. 6A, the first image is an intraoperative image havingpoint 103, the second image is a preoperative image. The point 103 iscopied onto the preoperative image via the method as described in FIG. 5(e.g., by locating the point on the preoperative image). In FIG. 6A thepoint is an IOL axis mark. A first triangle is constructed havingfeatures 101 and 102, and point 103 as the vertexes. Features 101 and102 are the features in the first image from the received two featurepairs (e.g., as determined/selected from among a set of feature pairsfor the two images). Features 201 and 202 are the features in the secondimage from the received two feature pairs. Features 101 and 201 compriseone feature pair (i.e., they are matching features), and features 102and 202 comprise a second feature pair. A second triangle similar to thefirst triangle is constructed in the second image, as described abovewith respect to FIG. 5. In this manner, vertex 203 (e.g., the thirdvertex of the target triangle) in the second image is matched to point103 in the first image, or in other words in this manner point 103 isaccurately copied (e.g., transferred) from the first image to the secondimage.

FIG. 6B shows images of an example of copying a location between twoactual images of an eye, according to illustrative embodiments of theinvention. In FIG. 6B, the reference image is an intraoperative imageand the target image is a preoperative image. The intraoperative imageis sampled before IOL insertion and the reference point is arbitrarilychosen (e.g., three dots are depicted in this example to exemplify threeaxis marks). In this example multiple triangles are constructed forcopying a single location. In some embodiments, the resulting multipletarget locations are averaged to generate an averaged target location.In some embodiments, the multiple target locations can be filtered suchthat all outlier target locations are filtered out. The outlier targetlocations can be, for example, all target locations that are more than amaximum distance from an average target location. In some embodiments,feature-pairs that have been used to calculate the outlier targetlocations (e.g. by constructing similar triangles in the first andsecond images) are identified, and any target location that wascalculated based on these features pairs (e.g. and was not alreadyfiltered out) is also filtered out. This can improve the accuracy of theaveraged target location.

In some embodiments, the method described in FIG. 5 is modified suchthat it is not a triangle that is used to copy the at least onelocation, but another geometric relationship. The geometric relationshipcan be a relationship that is scale and rotation invariant. For example,similar triangles can stay similar irrespective of scale or rotation ofthe one image relative to the other. In various embodiments, thegeometric relationship is a mathematical formula. For example, amathematical formula that given three xy locations generates a fourth xylocation. In various embodiments, the geometric relationship can be acircle, square, hexagon, octogen and/or any geometric shape as is knownin the art. In some embodiments, more than two features are used tocreate the geometric shape. For example, three features may be used,together with the point, to generate a quadrilateral in the first image,and a similar quadrilateral may be constructed in the second image (e.g.based on the three corresponding matching features) to determine thematching point in the second image. In some embodiments, more than oneinstance of each shape is used. For example, an intersection of threecircles may define a location, each circle being defined by tworeference features and the reference location.

As described above, feature pairs and/or a set of feature pairs can bedetermined between two images. FIG. 7 is a flow chart showing a methodfor determining a set of feature pairs between features of two images,according to some embodiments of the invention. The method can involvereceiving, by a computing device, a set of feature pairs, each featurepair including one feature location from a first image and a matchingfeature location from a second image (Step 710).

In some embodiments, a first set of feature from a first image and asecond set of feature are received, and the set of feature pairs aredetermined. In these embodiments, the first set of features and/or thesecond set of features can be determined as is known in the art. Forexample, the first set of features and/or the second set of features inthe images can be detected by deep learning. The first set of featuresand/or the second set of features can be detected within theirrespective images and described by feature detection algorithms, as isknown in the art. Each feature can have a defined point location (e.g.,a location having sub-pixel resolution) and a descriptor, a vector ofvalues which describes the image patch around the interest point.

Features between two images (e.g. two images of the same scene) can bematched, for example based on their similarity (e.g., the similarity ofthe descriptors of the features). Features between two images may bematched even if the two images are not exactly the same. For example, ina cataract procedure, the preoperative image and the intraoperativeimage of the eye may be different, yet numerous matching image elementsare apparent in both images, such as elements in the iris and junctionsof blood vessels. As is known in the art, matching features based on thesimilarity of their descriptors does not guarantee that the matching isvalid. Indeed, in some cases many, or even the majority of matched pairsof features between two images are false. For example, when pairingfeatures from two different images of the same house, two featuresdetected at the corners of two different windows may be falsely paireddue to the similarity of the windows. Various algorithms are known inthe art for detecting features and for generating descriptors, such asSIFT, SURF, and other. Descriptors may also be generated via deeplearning, as known in the art.

The method can involve filtering, by the computing device, the set offeature pairs (Step 720). In some embodiments, the set of feature pairscan be filtered based on a predefined geometrical relationship betweenfeature locations from the set the feature pairs. For example, in somescenarios, a set of feature pairs can be received or some prior artmethods for feature matching can generate a list of feature pairs withfalse pairs. Filtering the set of features pairs can result in a set offeature pairs that has less features pairs then the received and/orgenerated set of feature pairs. Filtering the set of features pairs canresult in reducing false features pairs in the set of features pairs.

A set of feature pairs can be filtered by evaluating a geometricalrelationship between features. The geometric relationship can be scaleand rotation invariant. The geometric relationship can be based on aEuclidean distance (e.g., in pixels) between any two features in animage. The ratio of a reference distance (e.g., a distance between twofeatures in the reference image) to a matching target distance (e.g., adistance between the two matching features in the target image) can beapproximately equal to the relative scale of the two images. That is, ifany two pairs of features that were correctly paired are used tocalculate such a ratio, then the resulting ratio can be more or less thesame. By calculating the ratio for all the possible couples (orsubstantially all) of the pairs in the two images (e.g., for instanceusing all the pairs in the pair list that were not filtered out byearlier methods. e.g. methods based on the descriptor similarity score),and even without knowing the actual relative scale of the two images,the actual scale can stand out, for instance in a histogram of all theresulting ratios. Feature pairs that are used in the calculation ofratios around the correct scale may be identified and other featurepairs may be filtered out (e.g. discarded from the set of featurepairs). In some embodiments, additional filtering is performed based onother geometric relationships, for instance by using triplets of featurepairs and comparing the similarity of triangles they define. In someembodiments, the additional filtering is performed by using at leastthree feature pairs and comparing similarity of polygons they define inthe two images. In some embodiments, similarity can be similarity up toa predefined tolerance. The filtering can include discarding form theset of feature pairs at least one feature pair of the at least threefeature pairs when the polygons are non-similar.

In some embodiments, a geometric relationship for filtering outliers maybe for example based on the angle of the line connecting any twofeatures in an image. For example, the angle of the line connecting afeature in location (100, 100) and a feature in location (200, 100) is 0(e.g., where locations are in pixel values). The angle of the lineconnecting a feature in location (100, 100) and a feature in location(200, 200) is 45 degrees. The difference between a reference angle(e.g., an angle of a line connecting two features in the referenceimage) to a matching target angle (e.g., an angle of a line connectingthe two matching features in the target image) can be approximatelyequal to the rotation between the two images. In some embodiments, ifany two pairs of features that were correctly paired can be used tocalculate a difference of angles, then the resulting difference can bemore or less the same. By calculating this difference for all thepossible couples of pairs in the two images (e.g., using all the pairsin the pair list that were not filtered out by earlier methods, e.g.methods based on the descriptor similarity score), and even withoutknowing the actual rotation of one image compared to the other, theactual rotation can be visible in a histogram of all the resultingdifferences. Feature pairs that were used in the calculation of angledifferences around the correct rotation can be identified and otherfeature pairs can be filtered out. FIG. 8A shows how geometricalrelationship can be used to filter a set of feature pairs, according tosome embodiments of the invention. Five vessel junctions associated withfeatures 6010-6050 are illustrated in the reference image (top). Sixblood vessel junctions associated with features 6510-6560 areillustrated in the target image (bottom).

Assume a pairing algorithm (e.g., based on the similarity ofdescriptors) correctly paired the features to create a set of featurepairs as follows: 6010 and 6510, 6020 and 6520, 6030 and 6530, and 6050and 6550. Also assume features 6040 and 6560 were incorrectly paired,and thus is an false feature pair in the set of feature pairs. Inaccordance with the method described in FIG. 7, the set of feature pairscan be filtered. In some embodiments, an outlier filtering algorithm(e.g., algorithm to filter erroneous pairs) designed to detect erroneouspairs (e.g. false feature pairs) as described above can calculate thedistances between the various couples of features in each image, andcalculate the ratio of distances between matching pairs. For example, adistance between 6010 and 6020 in the reference image can be divided bythe distance between 6510 and 6520 in the target image, and the resultis 1.39. In this example, all of the calculated ratios based oncorrectly matched features are around 1.4, which is the relative scalebetween the two images. Ratios involving the outlier pair (6040, 6560)can generate other results. The algorithm may then conclude that thispair is incorrect and filter it out of the list of pairs. Note that theratios are not expected to be exactly 1.4 due to the various reasonsdiscussed earlier that render the use of a global fit non-optimal (e.g.distortions in the live image).

The features in the reference image and the target image areautomatically detected, and in general they are not necessarilyassociated with the center of a blood vessel junction, and/or they arenot associated with any image element that can be identified by a nakedeye.

FIG. 8B are diagrams that illustrate two histograms of ratios generatedfrom actual images of a retina, according to some embodiments of theinvention. The histograms peak at the relative scale in this case, about1.1. The majority of the histogram, excluding the peak, is generated byincorrectly paired features, and this distribution of ratios may beexpected for random incorrect pairing. The histogram can demonstrate thestrength of this filtering method, as it can allow to easily filter outthe incorrect pairs even when the number of inliers (e.g., correctpairs) is small, simply by identifying which pairs more frequentlycontribute to the histogram bins around the peak and which pairs morefrequently contribute to other histogram bins. The two histograms inthis example are generated based on the same two images. The initiallist of feature pairs can be generated by matching all the featuresdetected in one image with features in the second image based on findingthe feature with the most similar descriptor (e.g., first runner-up).The top histogram was generated after the initial list of feature pairswas initially filtered by taking only pairs where the first runner-updescriptor similarity score was significantly higher than the secondrunner-up descriptor similarity score (e.g., with a 0.9 score ratiothreshold).

In this example, the number of pairs after this filtering stage was1667. The total number of calculated distance ratios in the histogram isthus 1667*1666/2. Based on this histogram, 66 feature pairs wereidentified as the most frequent to contribute to the peak bins. In thisexample, the pair that contributed the most to the peak bin wasidentified, and only pairs that contributed at least 80% of this pair'scontribution survived the filtering. As mentioned above, furtherfiltering based on triplets of pairs may eliminate the very few outliersthat survive this stage, if it is required. In this example, only 2 werefurther filtered, leaving 64 verified pairs.

The bottom histogram was generated without any initial filtering. Allthe features that were detected in one image (in this case 9396features) are matched with features in the second image (7522 features,e.g., “double booking” was allowed), totaling with 9396 pairs. Based onthis histogram, 143 feature pairs were identified as the most frequentto contribute to the peak bins when using the 80% threshold as above,and 395 were identified when using a 50% threshold.

In various embodiments, the filtering of feature pairs can beimplemented either on the entire image or on separate image tiles. Whenused on the entire image, the peak in the histogram can be slightlywider than when used in tiles, due to fact that the two images are notperfectly identical (e.g. even if scale, rotation and translation wereto be corrected), as described earlier. Implementing the filteringmethod in tiles can involve generating several separate histograms usingfeatures belonging to several smaller areas (e.g., tiles) in thereference image (e.g., together the tiles may cover the entire image,and an overlap is allowed between tiles). Since local features (e.g.,features belonging to the same area in the image) can be assumed to beaffected by common image distortions, the histograms generated locallycan exhibit narrower peaks. In some embodiments, an initial set offiltered pairs is identified on the entire image, and are used to definecorresponding tiles between the images, for instance by calculating thecorresponding locations of tile corners in the second image (e.g., byusing the triangles similarity method as described above in FIG. 5).Once corresponding tiles are identified, features may be detected andpaired separately in each pair of corresponding tiles in both images,thus increasing the total number of verified pairs.

In some embodiments, a second feature pair search is performed within anarea of each filtered feature pair. In these embodiments, the filteringof the feature pairs in the second round can be less limiting than theoriginal filtering such that the total number of filtered pairs canincrease.

In some embodiments, any feature pairs comprising features that may beregarded as unreliable reference features for image-to-image locationmatching during a procedure are excluded from being included in thefeature pair set. For example during anterior segment ophthalmicsurgery, any feature pairs that are not part of the limbus, or within acertain distance of the limbus, may be regarded as unreliable referencefeatures for image-to-image location matching, and may be excluded frombeing included in the feature pair set.

In some embodiments for anterior segment ophthalmic surgery, bloodvessels are classified to scleral vs. conjunctival, and feature pairsthat are associated with conjunctival blood vessels are excluded fromthe feature pair set. A feature pair is associated with conjunctivalblood vessels for instance when one of the two features (e.g., thefeature from the live image) is within a predetermined distance of bloodvessels that were classified as conjunctival. The classification may beimplemented, for instance, based on image sequences at the initial stageof the surgery, e.g. by implementing image processing algorithms and/ordeep learning algorithms.

In some embodiments, once features are detected in both images and a setof filtered feature pairs is generated, the locations of the features inthe live image are tracked. Tracking features may be implemented forinstance by template matching between consecutive frames in the liveimage sequence, where the templates are small image patches around thefeatures, and the searching area in the current frame is limited to thearea around the feature location in the previous frame. In someembodiments, the process of detecting features and determining featurepairs is repeated during the procedure, and the features are trackeduntil a new set of feature pairs is determined. In some embodiments, theprocess of detecting features and determining feature pairs is repeatedduring the procedure at a predetermined frequency. Repeating thisprocess can eliminate or substantially eliminate errors in featuretracking.

In some embodiments, the set of feature pairs includes features thatcover an entire region of interest for a guided surgical procedure. Forexample, for an anterior segment ophthalmic surgery, in someembodiments, it can be advantageous that the features used are not allconcentrated in one side of the eye, and instead they are substantiallyuniformly distributed around the limbus area. A uniform coverage canguarantee that local triangles are constructed even when some parts ofthe limbus area are hidden, e.g. by tools, and/or when some parts aredistorted. e.g. due to hemorrhages or liquids on the eye.

In some embodiments, preoperative images exhibit an eyelid partiallycovering the limbus, rendering feature detection impossible in thecovered areas. This can decrease the robustness of the trianglesimilarity method during surgery. In some embodiments at the beginningof the procedure a “proxy” image can be saved. The proxy image can be anintraoperative image, saved prior to the beginning of the guidedprocedure. The overlay location can be copied from the preoperativeimage CS to the proxy image CS using the triangle similarity method(e.g. as describe above with respect to FIG. 5), and later in theprocedure the proxy image can be used as a reference image for copyingthe overlay to the live image. In this manner, guidance information canbe copied between two intraoperative images.

In some embodiments, the preoperative image is registered with the proxyimage and overlay location can be copied from the preoperative image CSto the proxy image CS based on the determined transformation between theimages. In some embodiments, using a proxy image can be advantageousalso in other scenarios (e.g., other than anterior segment ophthalmicsurgery).

FIG. 9 is a diagram of an example of a system 2000 for overlayingguidance information, according to some embodiments of the invention.The system can include a camera system 2100 (e.g. a stereoscopic camerasystem), a video processing unit 2200 (e.g. embedded hardware), aprocessor 2300 (e.g. a PC), a head wearable display (HWD) 2400, and amonitor 2500.

During operation, the video processing unit 2200 can receive and readimages from cameras in the camera system 2100, process the images, andstream the processed images to the HWD 2400 and/or the monitor 2500. Theprocessing of the images can involve standard image processing (e.g.de-Bayer, automatic gain control, distortion correction), adding imagesin PIP view or in side-by-side view, and/or overlaying guidanceinformation, e.g. as guidance symbols. In some embodiments, the monitor2500 is a 3D monitor. The 3D monitor can be viewed with special glassesto see a 3D image.

In some embodiments, the system 2000 includes up to three HWDs 2400 thatcan simultaneously display a live image. The images displayed via thethree HWDs can be the same or different. For example, a supervisingsurgeon may zoom out, freeze the image and/or use menus for drawingmarkings on the image, while at the same time the resident is viewingthe live image without any change. In some embodiments, the images havevarying magnifications, e.g. when the magnification is digital.

In some embodiments, the camera system 2100 is assembled on a standardsurgical microscope (not shown). In some embodiments, the camera system2100 replaces the microscope oculars. In some embodiments, the system2000 includes both the camera system 2100 and microscope oculars. Inthese embodiments, beam-splitters can be used to partially deflect theoptical images towards the cameras. In some embodiments, the camerasystem 2100 has a single camera. In some embodiments, the camera systemhas a stereoscopic camera (e.g. two cameras).

In some embodiments, when the camera system 2100 is assembled on astandard surgical microscope, in addition to or instead of overlayingthe guidance information on an image displayed via an HWD and/or amonitor, the guidance information may be superimposed on the opticalimage viewed through the oculars (e.g., beam-splitters may be used todeflect the overlay images towards the oculars). In these embodiments,the overlay images can include the guidance information on a blackbackground, such that only the guidance information is superimposed onthe optical image generated by the surgical microscope, and other areasin the overlay images do not obscure the optical image. The overlayimage (or images) can be generated based on the image (or images)captured by the camera (or cameras) of the camera system, for instanceusing the image-to-image location matching method as described abovewith respect to FIG. 5. The overlay images in these embodiments canrequire a correction for allowing the overlay to be accuratelysuperimposed on the optical image viewed via the oculars. The correctioncan be based on a predetermined alignment between the camera and thecorresponding optical image as viewed via the ocular (e.g. that takesinto consideration also the different optical distortions of the twochannels).

It should be noted that although image-to-image location matching isdescribed in the context of an ophthalmic surgical microscope system,the same may be implemented for other surgical microscope systems, suchas a brain surgical microscope system, and other surgical applications(e.g. non-microscopic) where guidance may be used in conjunction withlive video, such as endoscopic and laparoscopic surgery. In someembodiments, the image-to-image location matching can be used fornon-surgical applications.

FIG. 10 shows a block diagram of a computing device 1400 which can beused with embodiments of the invention. Computing device 1400 caninclude a controller or processor 1405 that can be or include, forexample, one or more central processing unit processor(s) (CPU), one ormore Graphics Processing Unit(s) (GPU or GPGPU), FPGAs, ASICs,combination of processors, video processing units, a chip or anysuitable computing or computational device, an operating system 1415, amemory 1420, a storage 1430, input devices 1435 and output devices 1440.

Operating system 1415 can be or can include any code segment designedand/or configured to perform tasks involving coordination, scheduling,arbitration, supervising, controlling or otherwise managing operation ofcomputing device 1400, for example, scheduling execution of programs.Memory 1420 can be or can include, for example, a Random Access Memory(RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a SynchronousDRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, avolatile memory, a non-volatile memory, a cache memory, a buffer, ashort term memory unit, a long term memory unit, or other suitablememory units or storage units. Memory 1420 can be or can include aplurality of, possibly different memory units. Memory 1420 can store forexample, instructions to carry out a method (e.g. code 1425), and/ordata such as user responses, interruptions, etc.

Executable code 1425 can be any executable code, e.g., an application, aprogram, a process, task or script. Executable code 1425 can be executedby controller 1405 possibly under control of operating system 1415. Forexample, executable code 1425 can when executed cause masking ofpersonally identifiable information (PII), according to embodiments ofthe invention. In some embodiments, more than one computing device 1400or components of device 1400 can be used for multiple functionsdescribed herein. For the various modules and functions describedherein, one or more computing devices 1400 or components of computingdevice 1400 can be used. Devices that include components similar ordifferent to those included in computing device 1400 can be used, andcan be connected to a network and used as a system. One or moreprocessor(s) 1405 can be configured to carry out embodiments of theinvention by for example executing software or code. Storage 1430 can beor can include, for example, a hard disk drive, a floppy disk drive, aCompact Disk (CD) drive, a CD-Recordable (CD-R) drive, a universalserial bus (USB) device or other suitable removable and/or fixed storageunit. Data such as instructions, code, NN model data, parameters, etc.can be stored in a storage 1430 and can be loaded from storage 1430 intoa memory 1420 where it can be processed by controller 1405. In someembodiments, some of the components shown in FIG. 10 can be omitted.

Input devices 1435 can be or can include for example a mouse, akeyboard, a touch screen or pad or any suitable input device. It will berecognized that any suitable number of input devices can be operativelyconnected to computing device 1400 as shown by block 1435. Outputdevices 1440 can include one or more displays, speakers and/or any othersuitable output devices. It will be recognized that any suitable numberof output devices can be operatively connected to computing device 1400as shown by block 1440. Any applicable input/output (I/O) devices can beconnected to computing device 1400, for example, a wired or wirelessnetwork interface card (NIC), a modem, printer or facsimile machine, auniversal serial bus (USB) device or external hard drive can be includedin input devices 1435 and/or output devices 1440.

Embodiments of the invention can include one or more article(s) (e.g.memory 1420 or storage 1430) such as a computer or processornon-transitory readable medium, or a computer or processornon-transitory storage medium, such as for example a memory, a diskdrive, or a USB flash memory, encoding, including or storinginstructions, e.g., computer-executable instructions, which, whenexecuted by a processor or controller, carry out methods disclosedherein.

One skilled in the art will realize the invention can be embodied inother specific forms without departing from the spirit or essentialcharacteristics thereof. The foregoing embodiments are therefore to beconsidered in all respects illustrative rather than limiting of theinvention described herein. Scope of the invention is thus indicated bythe appended claims, rather than by the foregoing description, and allchanges that come within the meaning and range of equivalency of theclaims are therefore intended to be embraced therein.

In the foregoing detailed description, numerous specific details are setforth in order to provide an understanding of the invention. However, itwill be understood by those skilled in the art that the invention can bepracticed without these specific details. In other instances, well-knownmethods, procedures, and components, modules, units and/or circuits havenot been described in detail so as not to obscure the invention. Somefeatures or elements described with respect to one embodiment can becombined with features or elements described with respect to otherembodiments.

Although embodiments of the invention are not limited in this regard,discussions utilizing terms such as, for example, “processing,”“computing,” “calculating,” “determining,” “establishing”, “analyzing”,“checking”, or the like, can refer to operation(s) and/or process(es) ofa computer, a computing platform, a computing system, or otherelectronic computing device, that manipulates and/or transforms datarepresented as physical (e.g., electronic) quantities within thecomputer's registers and/or memories into other data similarlyrepresented as physical quantities within the computer's registersand/or memories or other information non-transitory storage medium thatcan store instructions to perform operations and/or processes.

Although embodiments of the invention are not limited in this regard,the terms “plurality” and “a plurality” as used herein can include, forexample, “multiple” or “two or more”. The terms “plurality” or “aplurality” can be used throughout the specification to describe two ormore components, devices, elements, units, parameters, or the like. Theterm set when used herein can include one or more items. Unlessexplicitly stated, the method embodiments described herein are notconstrained to a particular order or sequence. Additionally, some of thedescribed method embodiments or elements thereof can occur or beperformed simultaneously, at the same point in time, or concurrently.

1. A method for determining, for a point in a first image, a matchinglocation in a second image, the first image and the second image are ofa patient, the method comprising: (a) receiving, via a computing device,the point having a location in the first image; (b) receiving, via thecomputing device, two feature pairs wherein each feature pair comprisesone feature location from the first image and a matching featurelocation from the second image; (c) determining, via the computingdevice, a first triangle with respect to the first image, such that thevertexes of the first triangle are the locations of the two features ofthe two feature pairs that are from the first image and the location ofthe point in the first image; (d) determining, via the computing device,a second triangle with respect to the second image, such that two of thevertexes of the second triangle are the locations of each of therespective matching features of the two feature pairs that are from thesecond image, and such that the second triangle has triangle similaritywith the first triangle, yielding a location of a third vertex; and (e)determining, via the computing device, the matching location in thesecond image of the point in the first image based on the location ofthe third vertex, wherein the first image and the second image areselected from at least one of: a preoperative image and anintraoperative image of the patient.
 2. The method of claim 1 whereinthe two feature pairs are selected from a set of feature pairs.
 3. Themethod of claim 2 wherein the two feature pairs are selected based ondistance of the two features of the two feature pairs that are from thefirst image to the point.
 4. The method of claim 1 wherein an order ofthe vertices in the second triangle proceeding clockwise is the same asthe order of the respective matching vertices in the first triangleproceeding clockwise.
 5. The method of claim 1 wherein each vertex anglein the second triangle is equal to the vertex angle of its respectivematching vertex in the first triangle.
 6. The method of claim 1 whereinthe images of the patient are images of an eye of the patient.
 7. Themethod of claim 6 wherein the images are of an exterior eye and thefeature locations are in the limbus area of the eye in the images. 8.The method of claim 6 wherein the images are of the retina of the eye.9. The method of claim 1 further comprises repeating the method of claim1 for each of multiple points, wherein the multiple points are generatedfrom guidance information that is to be overlaid on the second image andwherein the two feature pairs are updated for each of the multiplepoints.
 10. The method of claim 1 further comprising: repeating steps(b) through (d) for each of a plurality of feature pairs to determine aplurality of corresponding third vertex locations; calculating anaveraged third vertex location; and determining the matching location inthe second image of the point in the first image based on the averagedthird vertex location.
 11. The method of claim 1, further comprising instep (e), setting the location of the third vertex as the matchinglocation in the second image of the point in the first image.
 12. Themethod of claim 10, further comprising setting the averaged third vertexlocation as the matching location in the second image of the point inthe first image.
 13. A method for determining, for a point in a firstimage, a matching location in a second image, the first image and thesecond image are of a patient, the method comprising: receiving, via acomputing device, the point having a location in the first image;receiving, via the computing device, at least two feature pairs whereineach feature pair comprises one feature location from the first imageand a matching feature location from the second image, wherein each ofthe features in each feature pair has a corresponding location in itsrespective image; determining, via the computing device, a geometricalrelationship with respect to the first image, based on the locations ofthe at least two features pairs and the location of the point in thefirst image, wherein the geometrical relationship is scale and rotationinvariant; and determining, via the computing device, the matchinglocation in the second image based on the geometrical relationship andthe locations of the respective matching features of the at least twofeature pairs that are from the second image, wherein the first imageand the second image are selected from at least one of: a preoperativeimage and an intraoperative image of the patient.
 14. A method forfiltering a set of feature pairs between two images, the two images areof a patient, the method comprising: receiving, by a computing device, aset of feature pairs wherein each feature pair comprises one featurelocation from a first image and a matching feature location from asecond image; and filtering, by the computing device, the set of featurepairs based on at least one of a scale invariant and a rotationalinvariant geometrical relationship between feature locations from theset of feature pairs, wherein the two images are selected from at leastone of: a preoperative image and an intraoperative image of the patient.15. The method of claim 14 wherein the filtered set of feature pairs hasless feature pairs then the received set of feature pairs.
 16. Themethod of claim 14 wherein filtering the received set of feature pairsreduces false feature pairs in the set of feature pairs.
 17. The methodof claim 14 wherein the filtering further comprises: determining, by thecomputing device, a set of couples of feature pairs; determining, by thecomputing device, for each couple of the set of couples a first distancebetween the two feature locations of the two feature pairs of the couplethat are from the first image; determining, by the computing device, foreach couple of the set of couples a second distance between the twofeature locations of the two feature pairs of the couple that are fromthe second image; determining, by the computing device, for each coupleof the set of couples a ratio between the first distance and the seconddistance, yielding a set of respective ratios; and discarding featurepairs based on the set of respective ratios.
 18. The method of claim 17wherein the discarding further comprises: determining, by the computingdevice, a relative scale between the two images, based on the set ofrespective ratios; and determining, by the computing device, a selectedfeature pair to be discarded from the set of feature pairs based on thedetermined relative scale and based on the ratios that were determinedusing the selected feature pair.
 19. The method of claim 14 wherein thefiltering further comprises: determining, by the computing device, a setof couples of feature pairs; determining, by the computing device, foreach couple of the set of couples a first angle of a line connecting thetwo feature locations of the two feature pairs of the couple that arefrom the first image; determining, by the computing device, for eachcouple of the set of couples a second angle of a line connecting the twofeature locations of the two feature pairs of the couple that are fromthe second image; determining, by the computing device, for each coupleof the set of couples a difference of angles between the first angle andthe second angle, yielding a set of respective differences of angles;and discarding feature pairs based on said set of respective differencesof angles.
 20. The method of claim 19 wherein the discarding furthercomprises: determining, by the computing device, a rotation between thetwo images, based on the set of respective differences of angles; anddetermining, by the computing device, a selected feature pair to bediscarded from the set of feature pairs based on the determined rotationand based on the differences of angles that were determined using theselected feature pair.
 21. The method of claim 14 wherein the filteringfurther comprises: selecting at least three feature pairs from the setof feature pairs; determining, via the computing device, a first polygonwith respect to the first image, such that the vertexes of the firstpolygon are the feature locations of the at least three feature pairsthat are from the first image; determining, via the computing device, asecond polygon with respect to the second image, such that the vertexesof the second polygon are the feature locations of the at least threefeature pairs that arc from the second image; determining via thecomputing device, whether the first and second polygons are similar; andwhen the two polygons are not similar, discarding from the set offeature pairs at least one feature pair of the at least three featurepairs.